Gestural Attributions as Semantics in User Interface Sound Design

Sound-based communication within different kinds of media have a long tradition. Sound design of radio-plays as early as 1920s have defined the basis for communicative use of sound effects which is still relevant in the tradition of film sound design. One of the common demands for sound designers have been the creation of sound effects with various emotional or intentional qualities. The suitable door knock for narration, for example, can be described as urging, gentle or angry in the manuscript [1]. This kind of agency perspective for sound design is widely adopted in the tradition of film sound design exploiting the natural perceptual bias towards looking for cues of agency in actions. But when it comes to the sound design for human-computer interaction (HCI), this kind of expressiveness is seldom used. In this paper we argue that the design of non-speech audio feedback for the HCI context would vastly benefit from such agency approach, which aims at utilising the gestures of interpersonal interaction in the sonic interaction design. In the social interaction, we naturally express our mental states with bodily involvement which can be called either facial, vocal, hand or other kind of gestural action. As a communicative acts, gestures usually posses derived intentionality which is caused by the mental states of an acting person [2]. The involvement of gestural imagery in music perception[3], for example, suggests that similar motor-mimetic gestural attuning to the sound would be central element also in both the perception and the design of UI sounds. Therefore we suggest that gestural attributions of derived intentionality with sounds would be considered as a form of semantics and thus as means to design communicative sounds for interaction.